Skip to content

CKS: fix node drain on scale-down and upgrade, make a failed upgrade resumable - #14611

Open
bhouse-nexthop wants to merge 2 commits into
apache:4.22from
bhouse-nexthop:cks-node-drain
Open

CKS: fix node drain on scale-down and upgrade, make a failed upgrade resumable#14611
bhouse-nexthop wants to merge 2 commits into
apache:4.22from
bhouse-nexthop:cks-node-drain

Conversation

@bhouse-nexthop

@bhouse-nexthop bhouse-nexthop commented Nov 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Fixes how a CKS cluster drains a node on scale-down and upgrade, and lets a failed upgrade be retried and pick up where it stopped.

Fixes #14610

The drain fixes and the resumable upgrade are in one PR because they change the same upgrade loop in KubernetesClusterUpgradeWorker: the drain, the Ready wait, the per-node SSH address and the skip of finished nodes are all steps of that loop, and splitting them would mean two PRs conflicting on every line of it.

Drain (scale-down and upgrade)

Both workers had their own drain loop, and neither handled a drain that fails or takes a while. The most common cause is a PodDisruptionBudget that doesn't allow an eviction yet.

  • Scale-down gave up at once on a non-zero exit, counted an attempt twice after an exception, left the node cordoned, and sent the cluster to Alert. Nothing had been removed at that point.
  • Upgrade retried with no wait and left the node cordoned on the last failure. It now uncordons it, and also when it times out after the drain, unless the node was cordoned before the upgrade.
  • Both ran kubectl drain without --timeout, so a fixed 60s SSH timeout cut it off.

There is now one helper, KubernetesClusterActionWorker.drainKubernetesClusterNode:

  • Each attempt counts once. A non-zero exit or an exception is retried after a 30s wait, unless that would go past the operation's deadline.
  • It runs kubectl drain --timeout=<cloud.kubernetes.cluster.node.drain.timeout>s. The new setting defaults to 300 and is dynamic; values below 1 use the default.
  • The timeout is cut down when less time is left before the operation's deadline, so that pods aren't evicted by an attempt that can't be used. The SSH wait is the drain timeout plus 60s.
  • The error includes the end of kubectl's output, up to 2000 characters, as a drain blocked by a PodDisruptionBudget logs every retry.
  • The helper leaves the node as it is. The caller uncordons it, and only if the caller cordoned it.
  • Attempts: 3 on scale-down, as before; cloud.kubernetes.cluster.upgrade.retries + 1 on upgrade, as before.

Scale-down now cordons every node it is going to remove, then drains them one by one. A node that was cordoned already, for example by an operator, is left cordoned on a rollback. So is a node whose state can't be read. Cordoning them all first stops pods evicted from one being rescheduled onto another that is also going away.

  • If any node can't be cordoned or drained, every cordoned node is uncordoned. The cluster goes back to the state it was in, and the error names the node. Nothing has been removed, so Alert would be wrong.
  • After that, the nodes are deleted and their VMs destroyed as before. kubectl delete node gets its own retries, with --ignore-not-found so that a retry after a lost reply doesn't fail.
  • If the scale times out before every node is drained, it is rolled back in the same way, with nothing removed. Once the nodes are drained, the timeout no longer stops the removal part way. Before, a timeout could leave some nodes removed and the node count not updated.
  • If deleting a node or destroying its VM fails, the cluster goes to Alert as before. The drained nodes that are still in the cluster, and that were cordoned by the scale-down, are uncordoned, so they aren't left unschedulable.

A removed node's VM that can't be destroyed or expunged

  • Stale VM-map entry: once a node was deleted from Kubernetes, a failed destroy or expunge of its VM skipped the deletion of its kubernetes_cluster_vm_map entry, which then held the cluster in Alert for good (and Alert blocks scaling). The entry is now deleted whatever happens to the VM. An expunge failure no longer fails the scale-down; it is a warning event, and the expunge task retries the VM. A destroy failure still fails it, naming the VM to destroy by hand. If the scale-down stops part way, the cluster's node count, cores and memory are brought back in line with the map, and a VM left behind has the cluster's SSH port-forwarding rule revoked.

Upgrade

  • Ready check: every node is now waited on to be Ready after it is uncordoned, not only the first control node. Before, the next node could be drained while the previous one was still coming back.
  • SSH address: the script for node i was run on sshPort + i. That only works when the upgrade's VM list and the port-forwarding rules are in the same order. They aren't when there are manually upgraded nodes, which are left out of the list, or nodes added later, which get the next free port. The script could then run on the wrong node. Each node is now reached through its own SSH port-forwarding rule, the same lookup KubernetesClusterRemoveWorker already uses. Shared networks, where nodes are reached on their own address, are unchanged.
  • Error messages name the Kubernetes node as well as the VM.

Retrying a failed upgrade

A failed upgrade left the cluster in Alert. upgradeKubernetesCluster only accepted a Running cluster, so the upgrade could neither be retried nor finished. No schema change is needed to fix this.

  • Recording the upgrade: an upgrade now records the id of the version it is upgrading to as a cluster detail, upgradeKubernetesVersionId, not for display. That detail lives in the existing kubernetes_cluster_details key/value table. It is removed once the cluster is Running or Stopped again, for example because the upgrade succeeded or the Alert state scanner recovered the cluster, so it can't make a later, unrelated Alert look like a failed upgrade.
  • Accepting the retry: upgradeKubernetesCluster accepts a cluster in Alert when that detail is the requested version, or another version with the same semantic version (for example one registered with a fixed ISO), and all of the cluster's VMs are running. A version with the same semantic version as the cluster's own is never taken for a retry. There is a new transition Alert -UpgradeRequested-> Upgrading. Any other cluster in Alert is refused, with a message saying why.
  • Upgrades that failed before this change: these have no detail, so the nodes' versions are used instead.
    • The cluster's version only changes once every node has been upgraded.
    • Each node's version (kubernetes_cluster_vm_map.kubernetes_node_version) changes as soon as that node has been upgraded, uncordoned and found Ready. It is already shown as kubernetesnodeversion in the cluster's VM list, so progress can be followed there.
    • So a cluster still at the old version with a node at the target, not counting manually upgraded nodes, had an upgrade to that version stop part way.
  • Skipping finished nodes: a node is skipped when its recorded version is the target, kubectl get nodes also reports the target version for it, and it is Ready. The recorded value is never trusted on its own, and a node that went NotReady is upgraded again. A node that is Ready but was cordoned after its upgrade is skipped, and stays cordoned.
  • Exact version match: the existing check after each node is upgraded now uses the same comparison. It is exact, so 1.28.1 no longer matches 1.28.10, and it works line by line, because the SSH output also holds stderr, for example a warning from sudo. Semver build metadata (+vendor.1) is ignored.
  • Order: the first control node still comes first. If it was already done, it is skipped, and kubeadm upgrade apply doesn't run again. The other nodes only ever run kubeadm upgrade node.
  • UI: the Upgrade action is now also shown for a cluster in Alert.
  • Racing the Alert state scanner: the scanner tries to recover clusters in Alert, and could otherwise overwrite the state of an upgrade that has just started.
    • The upgrade now stops if it can't move the cluster to Upgrading.
    • reconcileAlertCluster no longer moves the cluster to Running unless it managed to move it to Recovering first.
    • A failed upgrade now detaches the binaries ISO before moving the cluster to Alert. Otherwise a quick retry could have its ISO detached by the job that failed.
    • These guards narrow the race, but they don't close it, because cluster state updates are not compare-and-set today (KubernetesClusterDaoImpl.updateState writes the new state without checking the old one). Stop concurrent CKS operations clobbering each other, recover stuck states #14609 adds compare-and-set state updates and stuck-state recovery, and closes this race. The two PRs are independent; whichever merges second needs a small rebase.

Overlap with #14604

#14604 ("Allow scaling a Kubernetes cluster that is in Alert") also changes KubernetesClusterScaleWorker.removeKubernetesClusterNode, to skip the drain for a VM that is down. Whichever of the two is merged second will need a small rebase. The skip moves into drainNodesToRemove here, so a down VM is neither cordoned nor drained.

The two don't depend on each other. When this PR rolls back a scale whose drain failed, a cluster that started in Alert goes back to Alert, not Running, so it stays correct once #14604 allows scaling from Alert. On 4.22 alone that branch is not reached.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

Unit tests only. This has not been run against a live cluster or hypervisor.

New tests:

  • KubernetesClusterActionWorkerTest:
    • the drain is retried after a non-zero exit and succeeds, with one wait and no uncordon;
    • an exception and then failures use every attempt and wait between them;
    • kubectl's timeout is cut down to the time left before the deadline, no attempt is made without time left, and there is no retry after an interrupted wait;
    • the exact kubectl drain --timeout=300s command and SSH wait are checked;
    • getNodeSshAddress uses the node's own port-forwarding rule on the cluster's public IP, ignores a rule that is being revoked, falls back to sshPort + index when there is none, and uses the private IP on a shared network.
  • KubernetesClusterScaleWorkerTest:
    • every node is cordoned before any is drained;
    • when the second node can't be drained, the first is uncordoned (the second already was, by the helper), the cluster gets OperationSucceeded (back to Running), not OperationFailed, and the error names the node and includes kubectl's output;
    • a cluster that started in Alert goes back to Alert;
    • a scale that is past its deadline drains nothing, uncordons the nodes and rolls back;
    • a node that was cordoned before the scale-down, as reported even with a line of stderr, is not uncordoned by it, and neither is a node whose state can't be read.
  • KubernetesClusterScaleWorkerTest, node removal: the happy path destroys, expunges and then drops each map entry; an expunge failure (false or an exception) drops the entry, records a warning and goes on; a destroy failure drops the entry, fails the scale, uncordons the nodes still in the cluster and takes the removed VMs off the node count, cores and memory; a destroy error for a VM that turns out destroyed or expunged goes on; a failed Kubernetes node delete keeps the entry; a failure after every node is removed still reconciles the counts; only the cluster's SSH rule on the removed VM is revoked.
  • KubernetesClusterUpgradeWorkerTest: a node is skipped only when both its recorded and its reported version are the target and it is Ready, also when it is cordoned. It is not skipped when only the recorded one is, when the node is NotReady, when the node can't be reached, when the recorded version is older, or when the versions only share a prefix.
  • KubernetesClusterManagerImplTest: the Alert retry check:
    • accepts the recorded upgrade version even when no node has reached it, and another version with the same semantic version, and refuses a different version;
    • without that record, accepts a node at the target version, and refuses a cluster with no node there or where only a manually upgraded node is there;
    • refuses a target that is the cluster's current version, or another version with the same semantic version.
  • KubernetesClusterUtilTest: the version match ignores a trailing newline, a line of stderr and build metadata, and is otherwise exact.

mvn -pl plugins/integrations/kubernetes-service -am test -Dtest='KubernetesCluster*Test' passes: 84 tests across 7 classes. Checkstyle passes on api and kubernetes-service.

How did you try to break this feature and the system with this change?

  • A normal upgrade from Running: every node is still at the old recorded version, so the skip check makes no SSH call and nothing changes.
  • Shared networks (SSH port 22): node addressing is unchanged.
  • Port-forwarding rules that are being revoked are ignored when looking up a node's SSH port.
  • An interrupted wait between drain attempts stops the retries instead of firing the rest with no wait.
  • A scale-down where every drain succeeds: nodes are deleted and their VMs destroyed in the same order as before. The difference is that all of them are cordoned and drained first.
  • Not covered: a scale-down that fails after some nodes have been removed still goes to Alert, as before. The nodes that are still there are uncordoned.

@bhouse-nexthop
CKS: fix node drain on scale-down and upgrade, make a failed upgrade …
0dcaa8f 
…resumable

Scale-down and upgrade both drain a node before touching it, each with its
own copy of the drain loop, and neither works well:

- Scale-down gives up at once when kubectl drain exits non-zero (a
  PodDisruptionBudget that does not allow an eviction yet, say), and when
  it does retry after an exception it counts the attempt twice. The node
  is left cordoned, and the cluster goes to Alert although nothing has
  been removed and every node still works.
- Upgrade retries straight away, with no wait for the eviction to become
  possible, and on the last failure leaves the node cordoned.
- In both, kubectl drain has no --timeout, so it is cut off by a fixed
  60s SSH timeout rather than a deadline of its own.

Both workers now use one drain helper in KubernetesClusterActionWorker:
one count per attempt, a retry (after a wait) on a non-zero exit or an
exception, kubectl drain --timeout from the new setting
cloud.kubernetes.cluster.node.drain.timeout (300s), cut down so that no
attempt runs past the operation's deadline, with the SSH wait derived
from it, and the end of the drain output in the error. The caller
uncordons the node, and only if it cordoned it. Scale-down cordons every node it is going to
remove (leaving alone the ones that were cordoned already), drains them, and if one cannot be drained uncordons them all and
puts the cluster back in the state it was in, with an error that names
the node, instead of Alert; the same happens if the scale times out
before every node is drained. Once they are drained, the removal is not
stopped part way by the timeout any more. If removing a node fails, the
nodes still in the cluster are uncordoned.

The upgrade had two more problems:

- It only waited for the first control node to be Ready, so the next
  node could be drained while the previous one was still coming back.
  It now waits for every node, after it has been uncordoned.
- It reached node i on sshPort + i. The port-forwarding rules are not
  created in the order of the upgrade's VM list (manually upgraded nodes
  are left out of that list, and nodes added later get the next free
  port), so the script could run on the wrong node. It now looks up the
  node's own SSH port-forwarding rule, as the remove worker already does.

A failed upgrade left the cluster in Alert with no way to finish it, as
upgradeKubernetesCluster only accepts a Running cluster. An upgrade now
records the version it is upgrading to as a cluster detail (an existing
key/value table) and removes it once the cluster is Running or Stopped
again. upgradeKubernetesCluster
accepts a cluster in Alert (Alert -> Upgrading) when that detail is the
requested version (or another one with the same semantic version), which
is a real upgrade of the cluster's own version,
and all of the cluster's VMs are running, and refuses
any other Alert cluster with a clear message. For an upgrade that failed
before this change, a cluster still at the old version with nodes
recorded at the target version (kubernetesnodeversion, which is set as
each node is done) is accepted too. Nodes that are already at the target
version, both as recorded and as reported by kubectl get node, and
that are Ready, are skipped. The first control node is still upgraded first; if it is
already done, kubeadm upgrade apply is not run again. The node version
check now compares versions exactly, so 1.28.1 no longer matches
1.28.10, line by line, as the SSH output can hold stderr too, and
ignoring semver build metadata. Failure messages name the node the upgrade stopped on. The upgrade
stops if it can't move the cluster to Upgrading, and the Alert state
scanner no longer marks a cluster Running when it couldn't move it to
Recovering, and a failed upgrade detaches its ISO before going to Alert,
so a retry and the scanner or the failing job don't undo each other's
work.

No schema change.

Signed-off-by: Brad House <bhouse@nexthop.ai>
@codecov

codecov Bot commented Nov 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.68085% with 114 lines in your changes missing coverage. Please review.
✅ Project coverage is 18.36%. Comparing base (d02adcd) to head (1370152).

Files with missing lines Patch % Lines
...er/actionworkers/KubernetesClusterScaleWorker.java 82.96% 22 Missing and 9 partials ⚠️
.../actionworkers/KubernetesClusterUpgradeWorker.java 40.38% 29 Missing and 2 partials ⚠️
...r/actionworkers/KubernetesClusterActionWorker.java 68.08% 23 Missing and 7 partials ⚠️
...bernetes/cluster/KubernetesClusterManagerImpl.java 43.75% 14 Missing and 4 partials ⚠️
...er/actionworkers/KubernetesClusterStartWorker.java 0.00% 2 Missing ⚠️
...ubernetes/cluster/utils/KubernetesClusterUtil.java 81.81% 1 Missing and 1 partial ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               4.22   #14611      +/-   ##
============================================
+ Coverage     18.32%   18.36%   +0.04%     
- Complexity    16644    16716      +72     
============================================
  Files          5939     5939              
  Lines        537768   538056     +288     
  Branches      65929    65979      +50     
============================================
+ Hits          98551    98831     +280     
+ Misses       428008   427985      -23     
- Partials      11209    11240      +31
Flag Coverage Δ
uitests 4.08% <ø> (ø)
unittests 19.46% <69.68%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bhouse-nexthop bhouse-nexthop mentioned this pull request Nov 8, 2026
Draft
2 of 9 tasks
@bhouse-nexthop

bhouse-nexthop commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Test report: CKS node drain on scale-down and upgrade, and resuming a failed upgrade

Environment

  • A CloudStack 24.0 (main) build with this PR, alongside some of my other open PRs, among them Allow scaling a Kubernetes cluster that is in Alert #14604 and Stop concurrent CKS operations clobbering each other, recover stuck states #14609.
  • The PR targets 4.22; it wasn't run on a 4.22 build. The scale and upgrade workers and the node scripts it changes are the same on both branches.
  • Three KVM hosts, Ceph RBD primary storage, one management server.
  • Two CKS clusters on isolated networks:
    • A: 3 control nodes and 1-3 workers;
    • C: 1 control node and 2 workers.
      Both went from Kubernetes v1.32.5 to v1.33.1 in the upgrade tests.
  • cloud.kubernetes.cluster.node.drain.timeout=60, set for the test (default 300). cloud.kubernetes.cluster.upgrade.retries=3.
  • The order of commands on the cluster comes from the first control node's sudo journal (COMMAND=/opt/bin/kubectl ...). The management server doesn't log a cordon or drain that succeeds.

Results

Scenario Result
S1: scale-down blocked by a PodDisruptionBudget pass: rolled back, nothing removed, an operator's cordon kept, back to Running
S2: scale-down with the PDB relaxed pass: every node cordoned first, then drained one by one, then deleted
S3: upgrade with VM order and SSH port order different pass: each node's script ran on that node; all 5 nodes on v1.33.1
S3b: "upgrade" to the version the cluster already has pass (unplanned): every node skipped, nothing run
S4a: upgrade fails part way (a worker stopped) pass: Alert, ISO detached first, target version recorded, node uncordoned
S4b: refusals in Alert pass: VMs not all running; a different version
S4c: retry from Alert pass: finished nodes skipped, the rest upgraded, Running on v1.33.1, detail removed

S1: drain blocked by a PodDisruptionBudget

  • Setup:
    • Cluster A was scaled to 3 workers, with a 4-replica web deployment spread over them and a PDB with minAvailable=4.
    • I cordoned one worker (C) by hand, as an operator would.
    • I then scaled to size=1, which removes workers B and C.
  • Order of commands on the cluster: 
    14:52:37 kubectl cordon C
14:52:37 kubectl cordon B
14:52:38 kubectl drain C --ignore-daemonsets --delete-emptydir-data --timeout=60s
14:54:08 kubectl drain C ... --timeout=60s
14:55:39 kubectl drain C ... --timeout=60s
14:56:39 kubectl uncordon B
    That is three attempts, each 60 s plus a 30 s wait. Only B was uncordoned. C, which was already cordoned, stayed cordoned.
  • The API error names the node and ends with kubectl's output: 
    Scaling failed for Kubernetes cluster ..., unable to drain Kubernetes node: <C> on VM: ...: ... Cannot evict pod as it would violate the pod's disruption budget.
...
error: unable to drain node "<C>" due to error: ... global timeout reached: 1m0s, continuing command...
. No node was removed
  • The cluster was Running with size 3, all three workers in Kubernetes, B schedulable and C still cordoned. The call took 244 s.
  • Nit: the 2000-character tail of kubectl's output starts mid-line (4f6ccb-fjf66" -n "default" (will retry after 5s): ...). Starting it at the next newline would read better.

S2: the same scale-down, with the PDB relaxed

  • I set minAvailable=2, uncordoned C, and scaled to size=1 again: 
    14:56:43 kubectl cordon C
14:56:44 kubectl cordon B
14:56:44 kubectl drain C ... --timeout=60s
14:56:46 kubectl drain B ... --timeout=60s
14:56:48 kubectl delete node C --ignore-not-found
14:56:59 kubectl delete node B --ignore-not-found
  • The cluster was Running with size 1. The deployment rolled out again on the remaining nodes.

S3: per-node SSH address during the upgrade

  • Isolated networks rebuild the SSH port forwards in VM order on every scale. To make the orders differ, I swapped the rules of the third control node and the worker by hand: 
    VM order: control-1  control-2  control-3  worker-1  worker-2
port:     2222       2223       2225       2224      2226
    The old code would have run the script for VM i on port 2222 + i. That is control-3's script on worker-1 and the reverse.
  • Upgrade v1.32.5 to v1.33.1: OK in 657 s. 
    Upgrading node control-1 ... Successfully upgraded   (15:01:18 - 15:05:21)
Upgrading node control-2 ... Successfully upgraded   (- 15:07:53)
Upgrading node control-3 ... Successfully upgraded   (- 15:10:08)
Upgrading node worker-1  ... Successfully upgraded   (- 15:11:11)
Upgrading node worker-2  ... Successfully upgraded   (- 15:12:03)
  • Every node had management server SSH logins during its own step. The first control node had 33 logins, because kubectl runs there. Every kubelet reports v1.33.1, which needs each node's own script to have run on it. All nodes were Ready and uncordoned. The cluster was Running on v1.33.1, and no upgradeKubernetesVersionId detail was left.
  • S3b (unplanned). My first attempt looked up the target version with a keyword filter, which picked v1.32.5, the cluster's own version. Stock only refuses a downgrade, so the request was accepted. Every node was found already at the target (recorded and in kubectl, and Ready), and skipped: 
    Skipping node control-1 / control-2 / control-3 / worker-1
    It returned in 9 s and nothing was run.

S4: a failed upgrade, then a retry from Alert (cluster C)

  • S4a. I started the upgrade to v1.33.1 and force-stopped the last worker once the first two nodes were on v1.33.1. 
    Upgrading node control ... Successfully upgraded
Upgrading node worker-1 ... Successfully upgraded
Upgrading node worker-2
Draining node: worker-2  16:01:43 / 16:03:14 / 16:04:44 / 16:06:15   (retries + 1 attempts)
Detached Kubernetes binaries from VM ... (all three)   16:06:16
-> "Failed to upgrade Kubernetes cluster : cluster-c, unable to drain Kubernetes node : worker-2 ..."
    • The cluster went to Alert, still on v1.32.5. The ISO was detached before that.
    • upgradeKubernetesVersionId was set to the v1.33.1 version's id.
    • worker-2 was uncordoned, because the upgrade had cordoned it.
  • S4b: refusals in Alert. 
    upgrade to v1.33.1 with worker-2 stopped:
  "Kubernetes cluster : cluster-c is in Alert state and not all of its VMs are running. Start them before retrying the upgrade"
(start worker-2)
upgrade to v1.33.13 (a different version):
  "Kubernetes cluster : cluster-c is in Alert state. It can be upgraded in that state only to retry an upgrade that failed, to the same Kubernetes version as that upgrade"
  • S4c: the retry to v1.33.1, OK in 112 s. 
    Skipping node control
Skipping node worker-1
Upgrading node worker-2 ... Successfully upgraded
    • The skipped nodes ran no upgrade script. The skipped worker had no management server login, and the control node's logins were the kubectl checks.
    • The cluster was Running on v1.33.1, with all nodes Ready on v1.33.1 and uncordoned. The detail was removed.

Result: pass.

  • Scale-down cordons all nodes first and rolls back cleanly, naming the node, when a drain can't finish. An operator's cordon is kept.
  • The upgrade reaches each node through its own port forward.
  • A failed upgrade can be retried from Alert, only to the same version and only with all VMs running, and it skips the nodes that are already done.

Not covered

  • The default drain timeout (300 s), and the timeout being cut short near the operation's deadline.
  • A scale-down that times out after the drain, and a failure deleting a node or destroying its VM (the Alert path).
  • An upgrade that failed before this change, with no detail (the node-version fallback).
  • Shared networks, where nodes are reached on their own address.
  • The UI's Upgrade action for a cluster in Alert.

This was referenced Nov 9, 2026
Open
Open
@bhouse-nexthop
CKS: drop a removed node's VM map entry even if its VM can't be expunged
1370152 
When a scale-down removes a worker, it deletes the node from Kubernetes,
destroys and expunges its VM, and only then deletes the VM's entry in
kubernetes_cluster_vm_map. If the destroy or the expunge failed, the
exception skipped that last step and the entry stayed, for a VM that was
Destroyed, or still running but no longer a Kubernetes node. Nothing
recovered from that: the state scanner moves an Alert cluster back to
Running only when every mapped VM is Running and Kubernetes reports as
many ready nodes as the cluster has, which a VM whose node is gone never
satisfies, and a cluster in Alert can't be scaled. The node count and
the next scale-down's choice of nodes were worked out from the stale
entry too.

The node leaving Kubernetes is now the point at which its VM leaves the
cluster:

- Once the node has been deleted from Kubernetes, the map entry is
  deleted after the attempt to destroy and expunge the VM, whatever its
  outcome. (Not before it: the VM's expunge looks the entry up, to leave
  alone the port-forwarding rules of a CKS node in an NSX zone.)
- If the VM can't be expunged, the scale-down goes on, with a warning in
  the log and a WARN event that names the VM and the state it was left
  in. The expunge task retries it (it picks up Destroyed, Expunging and
  Error VMs), or an administrator can. It no longer holds the cluster in
  Alert. Until it is expunged it keeps the cluster's network in use, so
  deleting the cluster with its network is refused, with an error that
  names the VM, as for any VM on the network that is not in the cluster.
  Nothing is deleted in that case, and the delete can be retried once
  the VM is gone.
- If the VM can't be destroyed, the scale-down fails, as it did, with an
  error that names the VM and says it is no longer part of the cluster
  and has to be destroyed by hand. (If the VM turns out to be Destroyed
  or Expunging anyway, it is expunged as usual and the scale-down goes
  on. A VM left in Error is not taken as destroyed, as it may still be
  running on its host.) Its entry
  is still dropped: keeping it would leave the cluster stuck in Alert as
  above, and a retry would try to cordon a node that Kubernetes no longer
  has. Without the entry the scanner can bring the cluster back to
  Running and the scale-down can be retried; the VM left behind is an
  ordinary VM of the account.
- A VM left behind either way has the cluster's SSH port-forwarding rule
  on it revoked, as expunging it would have done. The cluster's rule
  updates may not: none runs if the scale-down stops, and a VPC tier's
  only clears the remaining nodes' ports, while the rule, on a port
  beyond theirs, would conflict with the next node added. (A rule
  provided by NSX is left alone, as expunge leaves a CKS node's rules
  alone in an NSX zone, and so is the rule the scale-down reaches the
  cluster through, which the remaining nodes still need.)
- If the scale-down stops part way, for any reason, including a failure
  to update the network rules once every node is gone, the cluster's
  node count, cores and memory are reduced by the VMs whose entries were
  deleted, so that they match the map again (the scanner and the next
  scale compare the two). As in a full scale-down, which still sets
  them once at the end, every VM removed comes off the node count,
  including a control node removed by ID.

Signed-off-by: Brad House <bhouse@nexthop.ai>
@bhouse-nexthop

bhouse-nexthop commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Added a commit, 1370152 ("CKS: drop a removed node's VM map entry even if its VM can't be expunged"), for a related scale-down bug in the node-removal step this PR already touches.

When a node had been deleted from Kubernetes but its VM then failed to destroy or expunge, the exception skipped the deletion of the VM's kubernetes_cluster_vm_map entry. That stale entry never matches a Kubernetes node again, so the state scanner kept the cluster in Alert for good (and Alert blocks the next scale). The next scale-down's node count and node choice also worked from it.

Now the entry is deleted once the node is out of Kubernetes, whatever happens to the VM:

  • an expunge failure is a warning event and the scale-down goes on; the expunge task retries the VM;
  • a destroy failure still fails the scale-down, with an error naming the VM to destroy by hand;
  • if the scale-down stops part way, the cluster's node count, cores and memory are brought back in line with the map.

The commit message has the details, and there are unit tests for the happy path, expunge failure, destroy failure and partial failure. As before, I have only run unit tests, not tested against a live cluster.

@bhouse-nexthop

bhouse-nexthop commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Test report: scale-down when the removed node's VM can't be expunged or destroyed (commit 1370152)

Summary: a normal scale-down still drains, removes and expunges the node. When the removed node's VM can't be expunged, the scale-down now completes: it records a WARN event naming the VM, drops the VM's map entry, and fixes the node count, cores and memory. The cluster goes back to Running, not Alert. Once the blocker is gone, the expunge task removes the VM. When the VM can't even be destroyed, the scale-down fails with an error that names the VM. Its map entry is still dropped, so the cluster stays Running and a retry works.

Environment

  • A CloudStack 24.0 (main) build with this PR, alongside some of my other open PRs. The management server and all three agents were upgraded in place to this build.
  • The PR targets 4.22; it wasn't run on a 4.22 build. KubernetesClusterScaleWorker and the expunge code it depends on are the same on both branches.
  • Three KVM hosts, Ceph RBD primary storage, one management server.
  • A fresh CKS cluster, Kubernetes v1.32.13, on an isolated network. It had 1 control node and 1-2 workers, and autoscaling was off.
  • expunge.delay=60, expunge.interval=60.

Results

Scenario Result
R: normal scale-down, 2 -> 1 worker pass, 26 s. The node is removed and the VM expunged. The map entry is gone, counts are right, and the cluster is Running
E: scale-down with the removed VM's expunge blocked pass, 311 s. Completes, WARN event, map entry gone, counts right, Running. The VM is left in Expunging
E2: expunge blocker removed pass. The expunge task removed the VM 52 s later, and its root volume is Expunged
D: scale-down with the removed VM's destroy blocked pass. The scale fails with an error that names the VM. The map entry is dropped, counts are right, and the cluster is Running
D2: retry after D pass. size=1 returns OK and the cluster stays Running

How the failures were caused

  • Expunge (E). I held the VM's lock-table lock: a row in op_lock with key user_vm<vm id>, owned by a management server id that doesn't exist. This is what a concurrent operation on the VM does. UserVmManagerImpl.expunge() takes that lock first, and gives up after its 300 s wait, so it returns false. The VM itself is untouched, and deleting the row undoes it.
    • Why not the RBD route. I didn't make the root volume's RBD image undeletable (a protected snapshot or a watcher). VolumeOrchestrator.cleanupVolumes() only logs a volume that fails to expunge, so the VM's expunge would still succeed, and this code path wouldn't run.
  • Destroy (D). I assigned the VM to a backup offering of the dummy backup provider and took one backup. destroyVm(id, expunge=true) refuses a VM that has backups ("This Instance ... has a Backup Offering ... with 1 backups"). I had planned this as an expunge blocker, but the check runs in the destroy as well, so it covers the destroy failure instead. Deleting the backup undoes it. The backup framework was enabled for the test and turned off again afterwards.

R: normal scale-down

  • The last worker was removed. The cluster went Running -> Scaling -> Running, with size 1: 
    before: node_count=2 cores=6 memory=12288  map rows=3  k8s nodes=3
after:  node_count=1 cores=4 memory=8192   map rows=2  k8s nodes=2
removed VM: Expunging, removed set 11 s after the "Removing vm" line; map rows for it: 0
  • Events: Scheduled, Started, Completed ("Successfully completed scaling"). The log had no ERROR lines.

E: expunge fails

  • The worker was scaled back up to 2, and the new worker was the one to be removed.
  • The cluster went Running -> Scaling -> Running (size 1), and the call returned OK after 311 s. Most of that is the expunge's 300 s lock wait.
  • Log: 
    00:13:06 INFO  Removing vm ... i-2-487-VM ... from cluster pin-as
00:18:06 WARN  Unable to expunge VM ... i-2-487-VM ... of the node removed from Kubernetes cluster pin-as ...
  • Events: 
    KUBERNETES.CLUSTER.SCALE  WARN  Completed  Unable to expunge VM ... i-2-487-VM ... of the node removed from Kubernetes cluster ...
                                           The VM is no longer part of the cluster and is left in state Expunging, for the expunge
                                           task to retry. Until it is expunged it keeps the cluster's network in use, and the cluster
                                           can't be deleted with its network
KUBERNETES.CLUSTER.SCALE  INFO  Completed  Successfully completed scaling Kubernetes cluster
  • Afterwards: 
    cluster: Running, size 1   node_count=1 cores=4 memory=8192   map rows=2   k8s nodes=2
removed VM: Expunging, not removed; map rows for it: 0
its SSH port-forwarding rule (2224 -> 22): gone
  • Seven minutes later: the cluster was still Running with the same counts, so the state scanner didn't move it to Alert. The expunge task had tried the VM twice and logged Unable to expunge both times, since the lock was still held.

E2: blocker removed

  • I deleted the op_lock row. 52 s later the expunge task had expunged the VM (removed set), and its root volume was Expunged. The cluster was unchanged.

D: destroy fails, then a retry

  • The scale-down took 10 s. The call failed with this error: 
    Scaling Kubernetes cluster ... pin-as ... failed, unable to destroy VM ... i-2-486-VM ... (left in state Stopped).
Its node was removed from the Kubernetes cluster and the VM is no longer part of the cluster; it may still be
running and has to be destroyed manually
    The log has the cause: This Instance [...] has a Backup Offering [...] with 1 backups.
  • The cluster went Scaling -> Running; a failed scale returns to Running in the state machine. The map entry was dropped, and the counts already matched the map: node_count=1 cores=4 memory=8192, 2 map rows, 2 Kubernetes nodes. It was still Running 3 minutes later.
  • The VM left behind was Stopped. listVirtualMachines showed it with no Kubernetes cluster, and its SSH port-forwarding rule had been revoked.
  • Retry: scaleKubernetesCluster size=1 returned OK, and the cluster stayed Running. I then deleted the backup and destroyed the VM with expunge, which worked.

Result: pass. A removed node's VM that can't be expunged no longer leaves a stale map entry or holds the cluster in Alert. The scale-down completes with a WARN event, and the expunge task cleans the VM up once it can. A VM that can't be destroyed fails the scale with a clear error, and the cluster can still be scaled.

Not covered

  • Deleting the cluster with its network while a removed VM is still waiting for expunge. The commit says this is refused with an error that names the VM. I didn't try it here.
  • An NSX zone, and VPC tiers.
  • The 7-minute wait in E covered two passes of the expunge task. It didn't include a management server restart.

This was referenced Nov 10, 2026
Open
Open
@DaanHoogland DaanHoogland linked an issue Nov 11, 2026 that may be closed by this pull request
CKS: node drain on scale-down and upgrade fails too easily, and a failed upgrade can't be resumed #14610
Open
@DaanHoogland DaanHoogland added this to the 4.22.3 milestone Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

component:api component:kubernetes component:UI

Projects

None yet

Milestone

4.22.3

Development

Successfully merging this pull request may close these issues.

CKS: node drain on scale-down and upgrade fails too easily, and a failed upgrade can't be resumed

2 participants

@bhouse-nexthop @DaanHoogland